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EXECUTIVE  SUMMARY 


The  purpose  of  this  Central  Research  Project  (CRP)  is  to  gain  insight  into  the 
affect  situational  awareness  (SA)  has  on  small  unit  engagements  and  to  demonstrate  the 
use  of  human-in-the-loop  (HITL)  distributed  simulation  as  an  investigative  tool.  To 
accomplish  these  goals,  we  conducted  a  two-phased  simulation-based  experiment 
consisting  of  24  trials  per  phase  in  the  IDA  Simulation  Center  (SIMCTR).  During  the 
experiment,  distinct  situational  awareness  conditions  were  imposed  on  the  participants. 
Measures  of  effectiveness  were  computed  for  each  trial  and  inferences  were  made 
regarding  the  impact  of  situational  awareness  based  on  these  measures. 

Trials  consisted  of  simulated  company-level  armor  engagements  in  which  two 
opposing  teams.  Red  and  Blue,  attempted  to  destroy  objectives  belonging  to  the  opposing 
team.  Each  team  consisted  of  two  operators  controlling  16  Ml  main  battle  tanks 
simulated  by  Modular  Semi-Automated  Forces  (ModSAF)  version  4.0.  The  operators  on 
a  given  team  were  co-located  and  cooperated  tactically  during  each  of  the  trials. 
Opposing  sides  were  located  in  separate  rooms  and  were  not  able  to  view  their 
counterparts’  workstations  or  overhear  their  conversations. 

Two  distinct  SA  conditions  were  examined.  The  first,  referred  to  as  "visibility 
level,"  was  the  ability  of  each  Blue  player  to  see  the  units  his  teammate  controlled  and  the 
opposing  vehicles  detected.  The  second  SA  condition  was  the  availability  to  the  Blue 
force  of  unattended  ground  sensors.  Half  the  games  were  conducted  with  Blue  operating 
at  the  "high"  visibility  condition.  In  the  remaining  half.  Blue  players  could  only  display 
the  forces  under  their  direct  control.  Also,  in  half  the  trials,  Blue  had  access  to 
unattended  ground  sensors;  in  the  remaining  half,  the  sensors  were  removed.  Red  always 
operated  under  the  low  levels  situational  awareness.  Thus,  Red  never  had  unattended 
ground  sensors  and  each  Red  operator  could  only  see  the  units  controlled  by  his 
workstation  (and  the  units  they  detected). 

In  each  of  the  trials  Blue  operated  under  one  combination  of  the  situational 
awareness  conditions.  Thus  four  combinations  of  test  conditions  were  possible.  Six 
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replications  of  each  trial  combination  were  conducted.  The  experimental  was  a  classical 
22  factorial  design. 

At  the  end  of  each  trial,  the  armor  exchange  ratio  was  computed  and  the  score  of 
the  engagement — the  difference  between  the  number  of  objectives  lost  by  Red  and 
Blue — was  recorded.  Games  were  considered  wins  for  Blue  if  the  score  was  positive, 
wins  for  Red  if  negative,  and  ties  otherwise.  At  the  end  of  the  experiment,  analysis  of 
variance  (ANOVA)  was  used  to  determine  significant  factors  and  interactions  with 
respect  to  exchange  ratios  and  point  scores.  Multinomial  logit  analysis  was  applied  to 
determine  significant  factors  and  interactions  with  respect  to  wins,  losses,  and  ties. 

Of  the  24  games  in  the  first  phase,  Blue  won  7,  Red  won  only  1,  and  the 
remaining  16  were  ties.  Blue  never  lost  when  it  had  the  advantage  with  respect  to  either 
SA  factor.  Because  of  the  large  number  of  ties,  however,  the  research  team  decided  that 
we  should  conduct  a  second  more  complex  phase  of  the  experiment.  This  second  phase 
would  have  more  objectives  (to  reduce  the  likelihood  of  ties)  and  involve  slight  changes 
in  the  initial  unit  locations  to  make  it  easier  to  unify  friendly  forces.  These  changes  were 
made  to  encourage  cooperation  between  teammates.  Additionally,  the  objectives  were 
moved  to  more  protected  positions  and  a  limited  number  of  sensors  were  placed  around 
each.  These  changes  were  made  because  we  felt  that  defending  objectives  against  small 
incursions  was  particularly  difficult  and  may  have  been  a  factor  in  causing  so  many 
drawn  games. 

In  the  second  phase.  Blue  won  10,  Red  won  8,  and  6  games  were  tied.  Thus  the 
goal  of  reducing  the  percentage  of  ties  was  attained — but  at  a  cost.  In  the  first  phase, 
outcomes  generally  reflected  test  conditions:  Blue  did  better  on  average  when  it  had  an 
SA  advantage.  This  was  not  the  case  in  the  second  phase.  Except  for  the  fact  that  Red 
never  won  when  Blue  enjoyed  both  SA  capabilities,  the  data  revealed  no  discemable 
pattern  associated  with  test  conditions,  trial  order,  or  teams.  Apparently  the  additional 
objectives  in  conjunction  with  the  defensive  sensor  fields  introduced  too  much  variability 
into  the  scenario  or  allowed  in  unforeseen  factors  that  negated  the  SA  advantages. 

The  analysis  of  variance  and  other  statistical  procedures  did  not  associate  any 
significance  to  the  principal  factors  in  this  experiment.  Certain  interesting  trends 
appeared  in  the  data,  however,  and  these  may  be  worth  comment.  First,  in  Phase  I  (where 
each  team  had  three  objectives  to  attain  and  three  to  defend),  players  felt  strongly  that 
unattended  ground  sensors  were  a  greater  asset  than  full  visibility  for  the  Blue  team.  The 
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reasons  offered  were  based  on  the  belief  that  sensors  allowed  Blue  to  follow  the  progress 
of  the  Red  force  and  anticipate  its  routes  of  ingress.  While  visibility  allowed  Blue  to 
coordinate  maneuvers  (and  provided  additional  information  about  Red  positions  to 
individual  players),  this  was  not  considered  a  critical  advantage  since  players  believed 
they  operated  largely  independently  of  one  another.  The  data  indicated  that  sensors  had 
the  greater  impact  with  respect  to  point  scores,  but  not  with  respect  to  game  outcome 
(win,  loss,  tie).  Neither  factor  was  statistically  significant,  however. 

The  second  phase  (four  objectives  to  attain  and  defend)  is  much  harder  to 
interpret.  The  data  appear  to  suggest  that  the  imposition  of  an  additional  objective  added 
too  much  variation  to  the  games  for  the  principal  test  conditions  to  have  had  an  effect — 
except  when  both  factors  acted  in  concert  to  Blue’s  benefit.  Player  explanations  for  these 
outcomes  related  the  inability  of  Blue  to  outmaneuver  Red  and  defend  its  own  objectives 
within  the  timeframe  of  the  game.  The  advantage  given  Blue  by  only  one  test  condition 
may  have  been  insufficient  to  overcome  the  increased  burden  of  additional  objectives. 
When  Blue  had  at  most  one  advantage.  Blue  won  seven  games,  Red  won  eight,  and  three 
were  tied.  When  Blue  had  both  advantages.  Red  never  won:  Blue  won  three  games  and 
three  were  tied. 

Overall  conclusions  and  observations  regarding  the  experiment  appear  below. 
They  are  divided  into  two  parts:  observations  about  the  game  strategies  and  implications 
regarding  situational  awareness  experimentation.  Observations  regarding  the  play  of  the 
game  include  the  following  summary  of  tactics  and  players'  evaluation  of  factors 
affecting  situational  awareness: 

•  Emphasis  on  Attack — A  consistent  emphasis  was  on  attack  throughout  the 
trials.  Defense  was  difficult,  and  even  a  well-conducted  defense  might  result 
in  the  loss  of  a  defended  target  before  the  attackers  would  all  be  killed.  There 
was  also  a  kind  of  asymmetry  in  the  test  setup  that  inclined  the  operators  to 
attack.  A  battle  could  not  be  won  without  attacking  targets  in  enemy  territory. 
And,  although  the  battle  could  not  be  won  without  a  friendly  target  surviving, 
this  did  not  necessarily  mean  that  the  targets  needed  to  be  defended.  In 
particular,  if  an  attacking  force  met  and  defeated  an  enemy  attacking  force,  the 
enemy  might  be  unable  to  kill  all  friendly  targets  with  his  remaining  tanks, 
within  the  time  limit. 

•  Massed  Forces — A  larger  force  would  consistently  defeat  a  smaller  force, 
with  a  favorable  exchange  ratio  that  left  it  able  to  defeat  a  subsequently 
encountered  smaller  force. 
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•  Selective  Defense — The  perceived  need  to  mass  forces  and  to  attack  led  to  a 
resource  allocation  problem:  insufficient  forces  to  mass  defenders  at  each 
site.  In  many  cases,  some  or  even  all  friendly  target  sites  were  left 
undefended. 

•  Attack  with  Individual  Tanks — While  a  numerical  superiority  was  a  great 
advantage  for  tank  encounters,  only  a  single  tank  was  required  to  defeat  an 
undefended  enemy  target.  A  common  tactic  was  to  send  individual  tanks  to 
attack  targets,  in  case  they  were  not  defended. 

•  Regarding  Situational  Awareness  Factors,  the  Players’  General  Assessment 
Was 

—  There  was  an  advantage  to  having  sensors,  but  it  was  of  more  importance 
to  mount  a  planned  attack  rather  than  relying  on  sensor  data  to  provide  a 
decisive  advantage. 

-  Visibility  is  a  convenience  but  not  a  decisive  factor.  In  contrast  to  sensors, 
commanders  did  not  plan  their  tactics  to  take  advantage  of  visibility  and 
could  recall  few  instances  where  it  was  a  perceived  factor. 

Overall,  this  experiment  did  not  demonstrate  that  having  an  advantage  with 
respect  to  situational  awareness  yields  a  significant  benefit  in  small  unit  armor 
engagements.  As  this  report  suggests,  time  constraints  and  artifacts  of  the  simulation 
may  have  driven  outcomes.  Chapter  IV  outlines  these  issues  and  discusses  their  effect  on 
the  conduct  of  the  games.  If  this  is  indeed  the  case,  then  redesigning  the  experiment  with 
these  factors  in  mind,  and  possibly  selecting  a  different  simulation  [e.g..  Joint  Conflict 
and  Tactical  Simulation  (JCATS)]  might  show  different  results. 

On  the  other  hand,  one  could  argue  that  sufficiently  many  replications  of  the 
experiment  were  conducted  to  "average  out"  random  anomalous  behavior  in  the 
simulation  (one  reviewer  took  exception  to  this  explanation).  Also,  with  so  many 
replications,  consistent  artifacts  would  likely  have  been  recognized  and  successfully 
"gamed"  by  the  players.  If  this  is  indeed  the  case,  then  the  experiment  calls  into  question 
the  value  of  enhanced  situational  awareness.  Continuing  this  line  of  argument,  the 
experiment  may  be  demonstrating  that  either: 

1 .  The  S A  advantage  must  be  overwhelming  (such  as  one  side  having  complete 
knowledge  of  its  opponent’s  position  and  strength). 

2.  There  is  little  benefit  from  the  information  provided  by  enhanced  SA  without 
the  corresponding  experience  and  discipline  to  make  decisions  based  upon  the 
additional  information. 
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I.  BACKGROUND 


A.  SITUATIONAL  AWARENESS  AND  THE  TWO  SELECTED  FACTORS 

1.  Basic  Issues 

This  CRP  investigates  the  effect  of  situational  awareness  (SA)  on  the  outcomes  of 
small  unit  engagements.  The  notion  of  situational  awareness  as  used  in  this  context 
means  the  knowledge  a  commander  has  of  the  forces  under  his  or  her  control,  the 
neighboring  own-side  forces  not  under  his  control,  and  the  enemy  forces  within  his  zone 
of  authority.  The  level  of  this  knowledge  was  varied  in  a  series  of  highly  structured 
human-in-the-loop  (HITL)  trials  in  which  opposing  forces  engaged  one  another  in  free- 
play  scenarios.  The  results  of  this  experiment  are  the  subject  of  this  paper. 

This  research  was  undertaken,  in  part,  to  gain  insight  into  the  advantages  that 
digital  technology  brings  to  the  modem  battlefield.  Although  manifestations  of  digital 
technology  deserve  attention,  our  focus  and  emphasis  is  on  situational  awareness  for 
several  reasons:  its  importance  in  the  conduct  of  modem  warfare;  the  potentially 
tremendous  influence  of  digital  technology  on  SA;  and  the  ease  with  which  factors 
influencing  SA  can  be  modeled  in  human-in-the-loop  simulations.  Thus,  this  research 
helps  to  further  investigate  the  combat  utility  of  digital  technology  that  began  in  1997 
with  the  Task  Force  XXI  Advanced  Warfighting  Experiment  (TFXXI-AWE).  That 
experiment  was  conducted  to  test  the  Army  hypothesis  that  digital  technologies,  properly 
integrated  with  doctrine,  will  increase  the  lethality,  survivability,  and  tempo  of  the  force. 

Results  of  TFXXI-AWE  indicated  that  lethality  was  not  enhanced1  (with  respect 
to  the  baseline  conventional  force),  survivability  was  greater  in  one  out  of  four  missions 
for  the  digital  unit,  and  movement  rates  were  comparable  for  digital  and  baseline  units 
[Ref.  1],  Because  of  the  ambiguous  results  of  TFXXI-AWE,  experiments  such  as  the  one 


1  Smaller  percentages  of  digital  force  Mis  and  M2s  participated  in  direct  fire  engagements,  but  they 
were  equally  effective  as  their  baseline  counterparts. 
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described  in  this  paper  are  important  to  the  extent  that  they  shed  light  on  the  combat 
utility  of  digital  technologies  and  help  address,  indeed  test,  the  Army  hypothesis. 

2.  Selected  Factors 

In  this  experiment  we  varied  the  level  of  situational  awareness  that  one  side 
(Blue)  enjoyed  in  a  force-on-force  ffee-play  engagement  with  an  identical  opposing  Red 
force.  The  situational-awareness  components  that  varied  involved  the  level  of  visibility 
that  the  players  on  the  Blue  had  of  their  own  force  and  the  availability  (to  Blue)  of 
unattended  ground  sensors.  These  factors  were  selected  because  it  was  believed  that  each 
would  have  a  measurable  effect  on  the  ability  of  the  players  ("commanders")  to  conduct  a 
successful  engagement.  For  example,  it  was  believed  at  the  start  of  the  experiment  that 
allowing  Blue  players  to  locate  all  of  their  own-side  units  would  enhance  Blue’s  ability  to 
coordinate  attacks  and  defensive  maneuvers.  Similarly,  it  was  anticipated  that  unattended 
ground  sensors  would  enable  Blue  to  locate  and  react  to  Red  incursions  into  Blue’s 
territory.  Also,  these  SA  components  were  relatively  easy  to  implement:  visibility  levels 
were  menu-driven  at  each  workstation;  sensors  were  implemented  through  a  stand-alone 
sensor  simulator  that  an  IDA  analyst  developed  and  that  we  will  describe  in  detail  later  in 
this  paper. 

Situational  awareness  factors  other  than  the  two  selected  could  have  been  studied 
along  with,  or  instead  of,  visibility  and  sensors.  For  example,  the  experiment  could  have 
focused  on  the  utility  of  graphic  command  overlays  or  the  value  of  a  certain  quality  of 
communications  in  providing  commanders  with  a  clear  (mental)  image  of  the  battlefield. 
The  selected  factors  were  chosen  in  part  because  they  contribute  to  the  most  basic  aspect 
of  S  A,  namely  a  real-time  image  of  battlefield  elements,  and  in  part  because  they  could 
be  simulated  using  the  available  tools.  And,  they  demonstrate  how  distributed  simulation 
can  be  used  to  address  SA  issues. 

B.  PREVIOUS  EXPERIMENTATION 

This  experiment  follows  a  similar  CRP,  conducted  a  year  ago,  which  measured 
the  impact  on  small  unit  engagements  of  two  command  and  control  (C2)  configurations 
and  levels  of  visibility  of  the  opposing  force.  The  same  structure  (factorial  design)  was 
adopted  for  this  exercise  and  the  same  underlying  simulation  was  selected — Modular 
Semi-Automated  Forces  (ModSAF).  Twenty-four  trials  were  conducted  using  the  same 
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group  of  players  who  participated  in  the  current  experiment.  Results  suggested  that 
simplified  command  and  control  was  a  modestly  significant  factor  with  respect  to  trial 
outcomes  and  favorable  exchange  ratios.  Visibility  of  the  opposing  force  tracked  with 
favorable  outcomes  and  ratios,  but  was  not  significant. 

This  year’s  CRP  introduced  more  vehicles  for  player  control,  but  reduced  the 
number  of  vehicle  types  [the  earlier  experiment  used  helicopters  tanks  and  infantry 
fighting  vehicles;  only  tanks  and  high-mobility  multi- wheeled  vehicles  (HMMWVs)  are 
used  in  the  current  exercise].  The  previous  experiment  had  a  single  mobile  objective;  the 
current  trials  have  multiple — but  stationary — objectives.  Also,  players  this  year  were 
assigned  particular  vehicles  to  control  from  the  beginning  of  each  trial,  whereas  last  year 
each  team  chose  how  to  divide  its  assets  amongst  its  two  members  (and  could  change 
these  assignments  during  a  trial). 

Last  year’s  trials  were  more  heavily  focused  on  command  and  control  and 
consequently  entailed  a  more  complex  C2  scheme  than  the  current  trials.  These  are 
centered  on  situational  awareness  exclusively  and  involve  SA  aids  not  used  in  last  year’s 
tests.  Specifically,  sensor  fields  were  added  to  the  terrain  to  provide  knowledge  of 
opposing  units’  positions  and  to  serve  as  aids  in  the  defense  of  (stationary)  objectives. 

Taken  together,  the  2  sets  of  tests  represent  72  human-in-the-loop  trials.  It  is 
hoped  that  they  provide  some  evidence  for  the  assertion  that  HITL-based  distributed 
simulation  is  a  valuable  tool  for  investigating  the  factors  that  affect  small  unit 
engagements. 
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II.  DESCRIPTION  OF  TESTS 


A.  OVERVIEW  AND  CONCEPTS 

As  described  in  Chapter  I,  this  project  investigated  the  effects  of  situational 
awareness  on  small  armor  unit  combat.  The  approach  was  to  use  man-in-the-loop 
Distributed  Interactive  Simulation  (DIS).  We  conducted  a  series  of  simulation  tests,  in 
which  we  presented  computer  operators  representing  unit  commanders,  with  battlefield 
information  corresponding  to  different  situational  awareness  levels. 

The  tests  were  designed  within  the  constraints  of  project  resources  and  the 
simulation  environment  with  the  objective  of  identifying  the  types  of  battlefield 
information  that  might  most  affect  battle  outcomes  and  that  might  be  seen  by  operators  as 
being  most  useful.  Most  of  the  variables  associated  with  battle  outcomes  were 
controlled,  leaving  commanders’  decisions  (when  presented  with  varying  situational 
awareness  levels)  and  random  events  as  the  primary  determinants  of  outcomes. 

The  basic  test  setup  was  to  use  two  opposing  teams  of  two  operators  each,  with 
each  operator  controlling  two  Ml  platoons  (four  tanks  in  each  platoon).  The  vehicles 
were  simulated  by  ModSAF,  a  DIS  simulator  capable  of  simulating  vehicles  individually 
or  in  aggregate  units.  Operators’  information  about  the  battlefield,  and  thus  his 
situational  awareness,  was  provided  by  the  ModSAF  two-dimensional  plan  view  display 
and  also  by  a  similar  display  of  sensor  information  generated  by  the  Comprehensive 
Mine  Simulator  (CMS).  The  terrain  was  a  section  of  the  Hunter-Liggett  terrain  database 
that  had  been  reflected  to  create  a  synthetic  symmetric  terrain,  and  the  battle  objectives 
(for  each  side)  were  to  destroy  enemy  targets  at  specified  locations.  These  factors  are 
discussed  in  more  detail  in  Section  B.  The  symmetry  of  terrain,  forces,  and  commanders’ 
objectives  reduced  the  scenario  realism  but  enabled  a  strong  focus  on  the  effects  of 
differing  levels  of  situational  awareness. 
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B.  DISCUSSION  OF  KEY  SIMULATION  FACTORS 


1.  Operator  Controls  and  Situational  Awareness  Levels 

In  all  of  the  tests,  there  were  two  operators  on  each  side  (Blue  and  Red),  with  each 
operator  controlling  two  Ml  tank  platoons  of  four  tanks  each.  Each  operator  had  a 
separate  plan-view  display  (PVD),  a  two-dimensional  map  showing  features  such  as 
roads  and  terrain  contours,  overlaid  with  icons  representing  each  tank  under  the 
operator’s  control.  Each  unit  had  a  menu  of  commands  associated  with  it,  and  the  basic 
method  of  controlling  a  unit  was  to  select  a  command  to  move  to  a  location  and  then 
click  the  location  on  the  map. 

In  addition  to  the  vehicles  under  the  operator’s  control,  the  PVD  showed  vehicles 
within  sight  of  one  of  the  controlled  vehicles.  Thus,  the  basic  awareness  level  consisted 
of  knowledge  of  the  commander’s  vehicles  and  other  vehicles  within  sight.  In  each  test, 
the  Red  force  commanders  had  this  basic  awareness  level.  Battlefield  information 
presented  to  the  Blue  force  commanders  varied.  Additional  information  included 
combinations  of  two  factors: 

•  "Visibility"  consisted  of  the  display  of  the  other  Blue  vehicles  (the  two 
platoons  controlled  by  the  other  Blue  commander)  and  the  vehicles  in  their 
sight. 

•  "Sensors"  consisted  of  a  separate  PVD,  visible  to  both  Blue  commanders,  that 
displayed  vehicles  within  sensor  fields.  The  sensor  fields  consisted  basically 
of  the  Blue  side  of  the  terrain. 

2.  ModSAF 

DIS  is  a  simulation  environment  in  which  individual  vehicle  movements  and 
interactions  are  simulated  in  real  time.  It  supports  crewed  vehicle  simulators;  for 
example,  the  Ml  tank  simulators  use  a  full  crew  of  four  in  compartments  replicating  the 
Ml  interior,  using  realistic  controls  and  simulated  three-dimensional  views  of  the 
battlefield. 

ModSAF  is  a  widely  used  DIS  simulator  for  a  large  variety  of  vehicles,  both 
ground  and  air.  It  allows  a  single  operator  to  control  many  vehicles,  either  as  a  unit  (e.g., 
a  platoon)  or  individually.  This  "workstation"  type  of  simulator  uses  a  map  display 
(rather  than  a  3D  image  that  might  be  seen  from  one  viewpoint)  and  controls  using  the 
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computer  keyboard  and  mouse.  The  operator  enters  commands,  such  as  a  command  to 
move  to  a  designated  geographic  point,  and  can  set  parameters  such  as  desired  speed. 
ModSAF  simulates  the  vehicle  movement,  e.g.,  its  specific  path  and  speed,  considering 
factors  such  as  terrain  and  the  vehicle’s  characteristics.  In  addition  to  simulating 
movement,  ModSAF  incorporates  "behaviors"  intended  to  reflect  the  actions  of  the 
vehicle’s  crew.  For  example,  specific  weapon-firing  events  are  controlled  by  ModSAF 
rather  than  being  directly  under  the  control  of  the  computer  operator. 

ModSAF  has  proven  to  be  powerful  and  useful  for  many  purposes.  The  concept 
for  Semi-Automated  Forces  (SAF)  originally  emphasized  its  role  in  simulating  an  enemy 
who  used  standard  doctrinal  behaviors,  and  friendly  forces  adjacent  to  those  comprising  a 
higher  fidelity  simulation  using  manned  simulators.  It  has  also  proven  useful  for  analytic 
studies  (such  as  the  one  reported  here)  and  larger-scale  exercises.  It  has  limitations, 
however,  that  derive  primarily  from  the  programmed  behaviors  and  from  the  operator’s 
ability  to  control  vehicles.  Limitations  noted  by  the  operators  in  these  tests  included: 

•  Behaviors — There  are  some  types  of  complex  behaviors  that  ModSAF  does 
not  provide  and  that  are  difficult  to  perform  using  the  available  ModSAF 
commands  and  tools.  Two  examples  are 

-  Finding  good  defensive  positions,  that  offer  partial  concealment  and  good 
line-of-sight  to  enemy  approaches 

-  Conducting  a  battle  involving  coordination  among  vehicles. 

•  Control  Difficulties — The  basic  control  mechanism  consisted  of  commanding 
a  platoon  to  move  to  a  specified  point.  More  complex  commands,  to  follow  a 
specified  path,  were  sometimes  given.  These  commands  did  not  enable 
complex  maneuvering,  such  as  coordinating  an  attack.  A  platoon  could  be 
"split"  and  commands  issued  to  individual  tanks,  but  when  this  was  done  there 
was  a  risk  of  operator  overload  as  well  as  difficulties  in  coordinating  the 
movements  of  the  tanks. 

In  addition  to  these  difficulties,  there  were  instances  of  anomalous  behaviors.  For 
example,  one  tank  in  a  platoon  would  fall  far  behind  or  even  move  in  a  reverse  direction. 
Or  a  tank  would  not  fire  at  an  enemy  for  no  apparent  reason. 


11 


3.  Symmetric  Analysis 

A  previous  IDA  effort  investigated  the  possibility  of  "symmetric  analysis,"  in 
which  experimental  variables  would  be  controlled  by  using  the  same  or  symmetric 
parameter  values  for  two  opposing  forces.  Some  parameters,  such  as  force  structures,  are 
easily  controlled  in  simulations.  With  digitized  terrain  it  was  relatively  easy  to 
manipulate  the  data  structures  to  produce  a  symmetric  area  by  "reflecting"  an  area  along  a 
line.  There  is  some  lack  of  realism  in  the  resulting  terrain,  e.g.,  potential  loops  in  roads 
and  rivers  and  unrealistic  abrupt  changes  along  the  line  of  symmetry.  Conceptually, 
perhaps  the  most  limiting  and  potentially  unrealistic  aspect  was  giving  the  two  forces 
symmetric  objectives,  with  knowledge  of  the  opposing  force’s  starting  conditions  and 
objectives.  These  considerations  were  analyzed,  and  small  demonstration  tests  were 
conducted  in  earlier  studies  with  the  conclusion  that  symmetric  analysis  could  be  a  useful 
analytic  tool  [Ref.  2]. 

This  study  uses  the  symmetric  terrain  that  was  constructed  for  the  earlier  studies, 
a  (reflected)  portion  of  Hunter-Liggett  with  few  apparent  anomalies.  Figure  1  shows  the 
terrain,  a  square  region,  20  km  on  a  side.  The  Blue  and  Red  tank  platoons  are 
approximately  12  km  apart  and  about  15  km  from  their  respective  objectives.  The 
objectives  are  distributed  over  the  upper  and  lower  portions  of  the  terrain  in  bands 
approximately  17  km  in  length.  Own-side  platoons  initial  positions  are  3-4  km  apart. 
The  Ml  tank  platoons  travel  slightly  under  30  kph  in  open  terrain,  thereby  requiring 
nearly  30  minutes  to  reach  objectives  when  unopposed  (operators  could,  and  often  did, 
increase  speeds  by  direct  intervention). 
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Figure  1.  Terrain  Layout  and  Initial  Unit  Positions 


4.  CMS  Sensors 

The  Comprehensive  Mine  Simulator  (CMS)  was  used  in  these  tests  to  provide 
simulated  sensor  data  to  ModSAF  operators. 

CMS  was  developed  at  IDA  originally  to  simulate  and  study  smart  mines,  later 
extended  to  simulate  most  types  of  landmines,  and  most  recently  extended  to  simulate 
various  alternatives  to  antipersonnel  landmines.  Primarily  for  demonstration  purposes,  it 
was  extended  to  include  stand-alone  sensors — "mines"  that  do  not  explode,  but  that  use 
the  landmine  proximity  algorithms  to  track  nearby  vehicles  and  report  the  results.  The 
simulated  sensors  are  not  based  on  actual  sensors,  but  instead  have  parameters  such  as 
range  versus  probability  of  detection.  The  results  are  shown  on  a  PVD  display  similar  to 


that  of  ModSAF.  For  these  tests,  the  sensor  displayed  a  vehicle  icon  at  the  detection 
position. 

5.  Scenarios 

Two  series  (or  phases)  of  tests  were  conducted  in  this  study,  using  slightly 
different  scenarios.  Each  of  them 

•  Used  symmetric  terrain. 

•  Used  four  platoons  of  four  Ml  tanks  on  each  side  (Blue  and  Red),  with  two 
platoons  controlled  by  each  commander/operator. 

•  Used  symmetric  objectives.  The  objective  for  each  side  was  to  kill  enemy 
stationary  vehicles  (HMMWVs)  at  fixed  sites,  while  protecting  the  friendly 
HMMWVs. 

•  Varied  the  Blue  force  situational  awareness,  with  runs  at  all  combinations  of 
no  awareness,  visibility,  and  sensors  (as  described  in  Section  B.l). 

The  two  series  (phases)  differed  in: 

•  The  starting  locations  of  the  tank  platoons. 

•  The  number  and  locations  of  the  objective  HMMWVs. 

•  The  use  of  small  sensor  fields  around  HMMWVs  in  the  second  series  to  aid  in 
their  defense. 

C.  TEST  SETUP  AND  CONDITIONS 

1.  DIS  and  the  IDA  Simulation  Center 

The  situational  awareness  tests  were  conducted  in  the  IDA  simulation  center.  In 
addition  to  its  convenient  location,  this  site  was  selected  because  of  the  availability  of  a 
large  number  of  Silicon  Graphics  workstations  and  a  local  area  network  over  which 
independent  ModSAF  simulations  could  interact.  Two  physically  separate  areas  of  the 
Simulation  Center  were  configured  so  that  opposing  teams  could  communicate  with  their 
own  members  without  being  overheard  by  their  opponents.  Also,  workstations  were 
positioned  so  team  members  could  maneuver  forces  under  their  own  command,  while 
accessing  the  sensor  simulation  at  the  same  time. 
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In  the  IDA  Simulation  Center,  ModSAF  simulations  on  separate  workstations 
interact  by  exchanging  protocol  data  units  (PDUs)  over  the  local  area  network.  These 
PDUs  carry  information  related  to  the  state  of  the  entities  being  simulated.  For  example, 
they  convey  the  position,  orientation,  and  speed  of  vehicles  as  well  as  the  types  and 
locations  of  impact  points  of  rounds  fired.  The  PDUs  provide  the  necessary  information 
to  enable  separate  and  distinct  simulations  to  interact  in  a  credible  manner. 

2.  Teams 

The  project  team  had  four  members  with  ModSAF  experience.  They  became  unit 
commanders  and  various  pairings  of  these  four  players  made  up  the  six  teams  in  the 
exercise.  Each  team-pair  remained  intact  for  eight  successive  games  to  accommodate  all 
combinations  of  test  conditions  (four)  as  Blue  and  as  Red.  After  completing  a  block  of 
eight  games,  the  players  were  shuffled  and  new  teams  were  formed. 

3.  Conduct  of  the  Experiments 

Games  were  typically  played  in  blocks  of  four  with  teams  alternating  between 
Blue  and  Red.  Thus  two  sessions  were  required  for  a  team  to  complete  all  possible 
combinations  of  situational  awareness  factors  as  Red  and  as  Blue.  Games  were  limited  to 
45  minutes.  This  time  span  was  selected  because  it  gave  players  time  to  implement  their 
tactics  without  allowing  the  games  to  degenerate  into  long  wars  of  attrition  (see  Chapter 
IV,  Section  B.l  for  a  discussion  of  the  ramifications  of  limited  game  time).  Requiring 
teams  to  alternate  between  Blue  and  Red  was  intended  as  much  to  keep  players  from 
becoming  bored  as  it  was  a  test  design  consideration  to  reduce  the  impact  of  learning. 

Games  began  with  each  team  conducting  its  own  strategy  session.  These  were 
generally  5-10  minutes  in  duration.  During  these  sessions,  players  would  decide  on  their 
tactics:  which  avenues  of  approach  to  take;  how  many  units  to  leave  in  the  rear  for 
defensive  purposes;  where  to  join  forces  with  their  team  member;  how  many  tanks  to 
commit  to  attacking  the  opponent’s  armor;  which  units  to  send  in  pursuit  of  the 
objectives.  Once  the  respective  sides  were  satisfied  with  their  plans,  an  umpire  would 
signal  the  start  of  the  game.  A  data  logger  would  begin  recording  PDUs  at  this  point, 
providing  a  permanent  game  record  that  could  be  reviewed  for  analytic  purposes. 

During  the  course  of  the  games,  the  umpire  would  move  between  the  two  teams 
and,  when  necessary,  lend  technical  assistance  to  the  players.  Typically  this  would  be  of 
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the  form  of  helping  to  resolve  simulation  anomalies,  such  as  vehicles  that  would  not 
move  or  screens  that  "froze  up." 


4.  Run  Matrices 

Each  team  played  together  for  eight  games,  four  as  the  Blue  team  and  four  as  the 
Red  team  (alternating  between  Red  and  Blue  in  successive  trials).  In  each  case,  they 
played  through  a  complete  replication,  in  (pseudo)  random  order,  of  the  four  possible  test 
combinations:  low  visibility/low  sensors;  low  visibility/high  sensors;  high  visibility/low 
sensors;  high  visibility/high  sensors.  Thus  the  experiment  was  conducted  as  a 
randomized  complete  block  22  factorial  design  where  each  block  was  a  complete 
replication  of  the  four  test  combinations,  characterized  by  the  particular  team  that  played 
Blue  for  that  replicate.  Structuring  the  experiment  in  this  fashion  facilitated  the  ANOVA 
computations  in  which  each  of  the  factors,  their  interaction,  and  the  variability  due  to 
team  differences  were  accounted  for.  A  total  of  24  runs  were  conducted  in  each  phase. 
The  run  matrix  for  Phase  I  appears  below  in  Table  1 . 


Table  1 .  Run  Matrix  for  Phase  I 
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Table  1.  (Continued) 


Run  Number 

Visibility  Level 

Sensor  Level 

Team 

17 

L 

L 

E 

13 

L 

H 

E 

19 

H 

L 

E 

15 

H 

H 

E 

14 

L 

L 

F 

20 

L 

H 

F 

16 

H 

L 

F 

18 

H 

H 

F 

The  second  phase  utilized  the  same  players  and  same  experimental  design.  The 
order  of  the  runs  was  somewhat  different,  however.  Again,  24  runs  were  conducted  in 
six  blocks  of  four  trials.  The  matrix  of  runs  appears  in  Table  2  below. 


Table  2.  Run  Matrix  for  Phase  II 
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Typically,  a  group  of  four  runs  would  be  executed  in  one  afternoon,  a  given  run 
lasting  between  35  and  45  minutes.  Teams  would  alternate  between  playing  Red  and 
Blue  in  large  part  to  avoid  boredom  among  the  players  (players  had  to  physically  move 
between  rooms  in  order  to  change  sides).  Two  afternoon  sessions  were  required  to 
complete  two  blocks  with  half  of  each  block  completed  in  one  session.  Both  phases  were 
conducted  under  this  format.  From  a  statistical  perspective  it  may  have  been  preferable 
to  keep  one  team  playing  Blue  all  afternoon  (and  one  team  playing  Red)  so  that  the 
games  with  a  given  block  in  the  tables  above  would  have  been  contemporaneous. 
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III.  RESULTS 


A.  PHASE  I 

The  outcomes  for  Phase  I  trials  are  presented  below  in  Tables  3  and  4.  The 
columns  indicate  how  many  games  each  side  won  under  the  various  test  conditions. 
Outcomes  were  determined  by  point  scores,  the  number  of  objectives  attained  by  Blue 
diminished  by  the  number  of  objectives  attained  by  Red.  The  sign  of  the  score 
determined  the  winner:  positive  scores  imply  Blue  is  the  winner,  negative  imply  Red, 
and  zero  indicates  a  tie.  Recall  that  in  Phase  I,  each  side  had  three  objectives  to  "capture" 
and  three  to  defend. 


Table  3.  Phase  I — Initial  Results 


(Visibility  -  Sensors) 

Blue  Win 

Red  Win 

Tie 

L-L 

0 

1 

5 

L-H 

1 

0 

5 

H-L 

2 

0 

4 

H-H 

3 

0 

3 

Table  4.  Phase  I — Final  Results 
(after  re-running  the  first  four  trials) 


(Visibility  -  Sensors) 

Blue  Win 

Red  Win 

Tie 

L-L 

0 

1 

5 

L-H 

2 

0 

4 

H-L 

3 

0 

3 

H-H 

2 

0 

4 

The  first  table  shows  game  results  after  the  initial  set  of  24  runs  was  complete. 
The  first  four  runs  in  this  set  were  rerun  because  regular  team  members  were  unavailable 
for  these  runs  and  substitute  players  were  used.  The  results  after  the  re-runs  appear  in 
Table  4.  While  both  tables  suggest  that  there  is  mild  dependence  on  the  test  conditions — 
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namely  some  benefit  to  having  sensors  or  high  visibility — neither  condition  was 
significant  at  the  5  percent  level. 

In  each  set,  Blue  won  five  games  when  it  enjoyed  full  visibility  and  four  games 
when  it  had  unattended  ground  sensors.  Games  were  tried  seven  times  with  visibility  and 
eight  times  with  sensors.  There  is  a  slightly  stronger  indication  of  synergy  between 
factors  in  the  first  set  (three  wins  and  three  ties  versus  two  wins  and  four  ties  when  both 
are  high).  The  remainder  of  this  section  will  focus  on  the  point  scores  and  related 
measures  and  their  relationship  to  the  test  conditions.  Results  will  only  be  given  for  the 
rerun  trials  and  references  to  the  initial  set  of  runs  will  be  omitted. 

Point  scores  for  Phase  I  trials  appear  below  in  Figures  2  through  6.  As  explained 
above,  these  scores  are  the  difference  in  the  number  of  objectives  taken  by  Blue  and  Red. 
The  first  graph  in  each  pair  indicates  score  of  each  game  delineated  by  test  condition:  LL 
connotes  low  levels  of  each  factor;  HL  indicates  high  visibility  and  no  sensors;  LH  is  low 
visibility  with  sensors;  HH  is  the  high  level  of  each  factor.  The  second  chart  indicates  the 
total  points  scored  under  the  specified  test  conditions.  In  the  latter,  each  vertex 
represents  the  sum  of  six  trials,  as  there  were  four  different  test  conditions  in  the  24  trials 
(and  the  experiment  followed  a  balanced  design). 
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Figure  2.  Point  Scores  vs.  Run  Number  (Final  Set) 


An  observation  about  Figure  2  made  by  one  of  the  reviewers  related  to  the  early 
tendency  of  Blue  to  win  with  the  high  level  of  both  assets.  This  early  advantage 
disappears  rather  rapidly,  suggesting  an  adaptive  process  on  the  part  of  Red.  While  this 
aspect  of  Red  play  was  not  investigated  explicitly,  the  impact  of  the  run  order  on  outcome 
is  addressed  in  the  ANOVA  tables  in  Appendix  A. 

By  reviewing  Figure  3,  it  appears  that  sensors  have  an  overwhelming  impact  on 
point  scores  when  visibility  levels  are  low.  While  there  is  clearly  some  benefit  under 
these  conditions,  it  may  not  be  substantial  as  neither  factor  nor  their  interaction  was 
significant  at  the  5-percent  level  (after  controlling  for  variability  due  to  teams).  ANOVA 
tables  for  all  phases  appear  in  Appendix  A. 


Figure  3.  Point  Scores  vs.  Test  Condition  (Final  Set) 

Exchange  ratios  by  trial  category  appear  in  Figure  4.  Recall  that  engagement 
victories  were  not  necessarily  a  goal  for  either  team.  If  it  were  possible  to  gain  an 
objective  by  sacrificing  a  platoon  of  tanks,  then  Blue  or  Red  would  readily  make  the 
sacrifice.  Nonetheless,  the  ratio  of  Red  to  Blue  armor  losses  is  presented  because  it  is  a 
popular  measure  and  one  in  which  interest  is  often  expressed. 
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Figure  4.  Exchange  Ratio  (Final  Set) 


It  is  interesting  to  note  that  the  exchange  ratio  is  less  favorable  for  Blue  when 
Blue  enjoys  greater  visibility.  ANOVA  tables  in  Appendix  A  indicate  that  none  of  the 
factors  has  a  significant  impact  on  exchange  ratios,  however. 

The  test  was  conducted  with  the  same  four  players  throughout  the  entire  exercise. 
These  four  players  were  divided  into  six  teams  denoted  A  though  F.  Team  A  played  B, 
Team  C  played  D,  and  E  played  F.  Table  5  presents  the  wins,  losses,  and  ties  for  three 
pairs  of  teams  (where  wins  and  losses  refer  to  the  first  team  in  the  pair). 


Table  5.  Team  Performance  (Final  Set) 


Team  Pairs 

Win 

Loss 

Tie 

A-B 

1 

2 

5 

C-D 

1 

2 

5 

E-F 

1 

1 

6 

The  six  teams  performed  fairly  uniformly  throughout  the  course  of  the 
experiment.  No  real  dominance  is  apparent  from  Table  5.  Because  the  initial  set  had 
team  members  on  the  "A"  team  who  did  not  participate  in  the  remainder  of  the  exercise, 
the  corresponding  table  for  the  initial  set  of  runs  will  not  be  presented. 

While  the  point  scores  seemed  to  indicate  that  the  test  conditions  were  generating 
results  along  the  lines  one  would  expect  (there  was  some  albeit  not  significant  Blue 
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improvement  with  sensors),  the  large  number  of  ties  was  discouraging.  These  ties 
seemed  to  indicate  that  the  game  was  not  sensitive  enough  to  the  changes  in  situational 
awareness  and  might  never  generate  pronounced  differences  in  outcomes.  As  this  was 
central  to  the  research,  the  scenario  was  changed,  the  game  was  made  more  complex,  and 
a  second  set  of  24  trials  was  begun. 

B.  PHASE  II 

The  new  game  consisted  of  one  additional  objective  on  each  side  and  a  slight 
repositioning  of  the  tank  platoons  (symmetry  was  preserved).  The  additional  objective 
was  added  to  decrease  the  probability  of  tied  outcomes;  it  was  thought  more  objectives 
would  make  it  less  likely  that  both  sides  would  end  up  with  the  same  score.  The  platoons 
were  repositioned  to  make  it  easier  for  the  players  on  a  given  team  to  cooperate  on 
maneuvers — thus  emphasizing  the  benefit  of  high  visibility.  And  finally,  sensors  were 
added  in  the  vicinity  of  each  objective  on  the  Blue  and  the  Red  sides.  The  point  of  these 
additional  sensors  was  to  enhance  the  ability  of  each  side  to  defend  objectives  from  their 
opponent. 

Intuition  with  respect  to  objectives  and  ties  proved  to  be  correct.  The  percentage 
of  ties  dropped  sharply  from  Phase  I  to  Phase  II.  But  the  number  of  Red  victories 
increased  even  more  dramatically  and  appeared  to  transcend  any  benefit  that  Blue 
realized  from  high  visibility  and  the  repositioning  of  the  tank  platoons. 

Blue  won  on  five  occasions  when  it  had  full  visibility  and  on  five  occasions  when 
it  had  sensors.  However,  Red  won  twice  when  Blue  had  visibility  and  four  times  when 
Blue  had  sensors.  Red  won  twice  when  Blue  had  no  SA  advantage.  In  Phase  I,  Red  only 
won  once.  The  discussion  of  why  the  Red  performance  is  far  better  in  Phase  II  than  in 
Phase  I  will  be  postponed  to  a  later  section.  For  the  moment,  we  will  focus  on  the  Phase 
II  results  and  comment  on  their  significance.  One  of  the  more  tantalizing  outcomes  in 
Table  6  is  the  failure  of  Red  to  win  when  Blue  had  both  the  sensors  and  high  visibility.  It 
appears  to  suggest  that  the  combined  effect  gives  Blue  an  advantage  that  Red  cannot 
overcome.  The  narrative  portion  of  this  paper’s  analysis  will  develop  the  idea  more 
completely.  Statistically,  the  effect  is  only  marginal,  however  (multinomial  log-linear 
analysis  indicates  that  the  interaction  between  outcomes  and  visibility  is  significant  at  the 
5.2-percent  level,  but  no  other  outcome  interaction  is  significant  below  the  10-percent 
level). 
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Table  6.  Phase  II  Outcomes 


(visibility  -  sensors) 

Biue  Win 

Red  Win 

Tie 

L-L 

3 

2 

1 

L-H 

2 

4 

0 

H-L 

2 

2 

2 

H-H 

3 

0 

3 

Figure  5  shows  the  point  scores  as  a  function  of  run  number  delineated  by  test 
condition.  Unlike  the  companion  pictures  in  Phase  I,  there  is  no  apparent  pattern  (i.e.,  a 
long  horizontal  line  due  to  ties)  and  the  dispersion  is  far  greater.  In  this  second  phase,  the 
test  conditions  were  weaker  indicators  of  outcome.  Some  of  the  more  plausible  reasons 
for  this  lack  of  association  will  be  discussed  in  the  next  few  sections. 
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Figure  5.  Point  Scores  vs.  Run  Number  for  Phase  II 

Point  scores  for  the  various  combinations  of  test  conditions  in  Phase  II  appear  in 
Figure  6.  These  scores  show  a  different  form  of  interaction  from  Phase  I,  but,  like  the 
earlier  phase,  suggest  the  existence  of  some  payoff  from  the  combined  effects  of  sensors 
and  visibility.  The  difference  between  the  two  phases  is  the  large  disparity  in  scores 
when  sensors  are  available  in  Phase  II — while  the  corresponding  disparity  in  Phase  I 
occurred  under  low  sensor  conditions. 
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Score 


Figure  6.  Point  Scores  vs.  Test  Condition  for  Phase  II 


The  ANOVA  tables  for  Phase  II  in  Appendix  A  indicate  that  neither  of  the 
principal  factors  or  their  interaction  is  significant  (the  p-value  for  the  interaction  was 
about  17  percent).  They  also  show  no  significant  difference  among  teams  or  measurable 
effect  due  to  the  order  in  which  runs  were  made. 

As  in  Phase  I,  the  exchange  ratios  (see  Figure  7)  show  no  significant  sensitivity  to 
test  conditions.  In  fact  they  are  slightly  closer  to  1 .0  for  all  test  conditions  than  the  early 
set  of  runs.  Somewhat  reassuringly,  however,  they  are  more  aligned  with  intuition  in  that 
higher  visibility  results  in  better  outcomes  for  Blue  and  sensors  are  an  enhancement,  at 
least  in  low  visibility  conditions.  Again,  differences  are  not  significant. 


Figure  7.  Phase  II  Exchange  Ratio  (Red/Blue) 
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Table  7  below  contains  the  number  of  wins,  losses,  and  ties  accrued  by  each  team. 
As  in  the  Phase  I  case,  the  data  are  presented  for  the  pairs  of  teams  that  played  one 
another.  Team  performance  appears  less  uniform  than  was  the  case  in  the  earlier  phase. 
Indeed  team  A  dominates  team  B  and  team  E  appears  stronger  than  F.  Multinomial  logit 
analysis  showed  no  significant  interaction  between  pair  and  outcome,  however 
(regardless  of  the  order  of  the  individual  teams  in  the  pairings:  e.g.,  B-A  instead  of  A-B, 
etc).  Also,  recall  that  the  ANOVA  for  point  scores  did  not  show  team  variability  to  be  a 
significant  factor. 


Table  7.  Phase  II  Blue  Team  Performance 


Team  Pairs 

Win 

Loss 

Tie 

A-B 

4 

1 

3 

C-D 

3 

4 

1 

E-F 

4 

2 

2 
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IV.  OPERATOR  ASSESSMENTS 


During  the  course  of  the  experiments  the  four  operators  changed  their  tactics  in 
response  to  their  perceptions  of  tactics  that  worked,  tactics  that  their  opponents  were 
likely  to  pursue,  ModSAF  characteristics  and  capabilities,  and  simply  things  they  wanted 
to  try.  The  operators  discussed  tactics  among  themselves,  particularly  team  members. 
The  periodic  changes  in  team  composition  thus  encouraged  the  development  of  tactics. 
Following  the  tests,  the  operators  discussed  their  assessments  of  what  had  been  learned, 
specifically  how  to  account  for  the  outcomes,  and  two  operators  reviewed  the  recorded 
data  files  for  most  of  the  runs  for  additional  insights.  The  following  sections  discuss 
operators’  assessments  of  factors  that  affected  battle  outcomes  or  tactical  choices. 

A.  SITUATIONAL  AWARENESS 

The  test  objective  was  to  assess  the  value  of  situational  awareness,  and  the  tests 
were  designed  so  that  the  only  functional  difference  between  the  two  sides  in  any  run  was 
a  different  level  of  situational  awareness.  The  operators  were  always  aware  of  the 
difference,  and  planned  tactics  accordingly. 

In  general,  the  operators  regarded  "visibility,"  the  sharing  of  data  between  the  two 
operators  on  one  side,  as  not  being  particularly  useful.  "Sensors,"  having  knowledge  of 
sensor  detection  of  tank  movements  on  the  friendly  side  of  the  battlefield,  was  regarded 
as  useful  and  to  a  significant  extent  affected  the  tactics  chosen  by  each  side.  At  a  few 
points  during  the  tests,  particularly  following  the  first  set  and  at  about  the  midway  point 
of  the  second  set,  the  operators  were  told  that  the  interim  results  showed  that  visibility 
was  a  more  important  advantage  than  sensors.  At  the  time,  the  operators  regarded  this  as 
an  anomaly  that  resulted  partly  from  chance  and  partly  from  inexperience  in  using  the 
sensor  data.  Initially,  there  was  a  tendency  for  Blue  to  attempt  to  use  sensor  data  by 
holding  his  forces  in  his  own  area,  where  sensor  data  were  available,  and  to  wait  for  Red 
to  attack.  The  game  dynamics  were  such  that  Blue  might  be  unable  to  engage  Red  with 
superior  force,  or  that  Red  would  be  able  to  kill  the  Blue  objectives  even  if  he  lost  more 
tanks  than  Blue.  By  the  end  of  the  tests,  the  general  assessment  was  that  there  was  an 
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advantage  to  having  sensors,  but  that  it  was  more  important  to  mount  a  planned  attack 
rather  than  to  rely  on  sensor  data  to  provide  a  decisive  advantage.  Sensor  data  could,  in 
some  cases,  be  used  to  help  defend  particular  sites,  and  in  particular  it  prevented  Red 
from  infiltrating  the  Blue  side  undetected. 

Post-test  discussions  and  the  review  of  test  recordings  expanded  this  assessment. 
The  review  indicated  that  chasing  the  enemy  was  seldom  successful,  and  that  there  were 
many  instances  where  Blue  did  this.  Typically,  Blue  forces  would  be  deployed  forward 
of  the  sites  it  was  defending,  Red  would  attack  along  a  path  not  directly  through  the  Blue 
force;  Blue  would  fail  to  interpose  forces  directly  in  Red’s  path  and  would  end  up 
chasing  Red.  At  least  one  operator  deliberately  tried  to  delay  intercepting  Red  with  the 
thought  that  there  would  be  an  advantage  in  attacking  Red’s  flanks.  Red  would  usually 
kill  the  objective,  and  sometimes  would  proceed  to  another  objective  with  Blue  still 
chasing.  Without  sensor  knowledge  (except  within  1  km  of  his  sites)  Red  defenders 
would  stay  close  to  its  defensive  sites  and  intercept  any  Blue  attackers.  This  tactic  was 
more  successful. 

The  effects  of  visibility  remain  a  mystery  to  the  operators.  Visibility  has  two 
aspects,  that  all  Blue  forces  are  visible  to  each  Blue  commander  and  that  all  Red  forces 
seen  by  any  Blue  vehicle  are  visible  to  each  Blue  commander.  Plausibly,  each  aspect 
could  aid  Blue  coordination.  However,  Blue  forces  attempting  to  coordinate  their  actions 
were  generally  in  sight  of  each  other  so  that  visibility  provided  little  additional  own- force 
information  (just  the  locations  of  the  out-of-sight  forces,  e.g.,  remote  defensive  forces). 
And,  commanders  without  visibility  used  voice  to  tell  their  partners  about  enemy 
sightings.  At  the  end,  the  operators  subjective  assessment  is  that  visibility  is  a 
convenience  but  not  a  decisive  factor.  In  contrast  to  sensors,  commanders  did  not  plan 
their  tactics  to  take  advantage  of  visibility  and  could  recall  few  instances  where  it  was  a 
perceived  factor. 

B.  DESIGN  ISSUES 

As  described  earlier,  these  tests  were  designed  to  control  as  many  variables  as 
possible,  while  examining  different  levels  and  types  of  situational  awareness.  The  tests 
were  also  constrained  by  the  simulation  environment  and  the  scope  of  the  effort.  Some  of 
the  results,  both  in  terms  of  the  effectiveness  measures  and  the  tactics  that  were 
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employed,  were  affected  by  the  test  design  in  unforeseen  ways.  Factors  identified  by  the 
operators  included: 

•  The  time  limit 

•  Effectiveness  measure 

•  The  sensor  field  coverage. 

1.  The  Time  Limit 

A  time  limit  of  45  minutes  was  used  for  all  trials.  It  was  selected  initially  as 
sufficient  to  allow  a  typical  run  to  be  played  to  completion,  and  this  was  generally  the 
case  for  the  first  set  of  runs.  In  this  set  each  side’s  forces  were  widely  separated  and 
there  were  few  attempts  to  unify  them,  so  that  in  most  runs  there  were  separate  battles  on 
the  east  and  west  parts  of  the  terrain.  There  was  more  than  enough  time  to  fight  a  battle 
and  reach  the  opposition  objectives.  In  fact,  a  common  tactic,  particularly  for  Red  when 
facing  a  situational  awareness  disadvantage,  was  to  wait  on  its  side  of  the  terrain  and 
launch  an  all-out  attack  when  there  was  just  enough  time  to  reach  the  Blue  objectives.  If 
Blue  used  all  its  forces  to  defend,  it  would  not  have  time  to  reach  the  Red  objectives. 
Thus  Blue  could  not  afford  to  use  all  its  forces  in  defense,  and  Red’s  attack  would  most 
often  be  successful.  A  common  outcome  was  a  tie,  with  all  objectives  killed. 

In  the  second  set  of  trials  there  was  one  more  objective,  the  objectives  were 
somewhat  harder  to  reach,  and  each  side’s  forces  were  not  widely  separated.  This  led  to 
more  tactical  variations,  most  of  which  involved  some  degree  of  unification  or 
coordination  of  forces.  In  many  runs,  a  side  would  attack  in  one  sector  and  attempt  for 
elements  of  this  single  attacking  force  to  reach  all  objectives.  There  was  not  enough  time 
to  do  this  in  a  deliberate  fashion  (by  gathering  a  unified  force  and  having  it  attack  each 
objective  in  turn)  and  many  runs  ended  with  a  race  against  time  to  reach  an  objective. 

Thus,  in  each  set  the  time  limit  resulted  in  "gamesmanship"  and  affected  tactics. 
In  the  second  set,  it  was  one  of  the  reasons  Blue  could  not  afford  to  wait  on  his  side  of  the 
terrain  to  take  advantage  of  the  sensor  field. 

2.  Effectiveness  Measure 

Although  tank  kills  were  recorded  and  are  presented  in  Chapter  III,  the  objective 
for  each  side  was  to  kill  the  opposition  stationary  HMMWV  targets  while  defending  the 
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friendly  targets.  This  led  to  a  willingness  to  sacrifice  tanks  and  tactics  such  as  breaking 
off  from  tank  battles  to  attack  the  objectives.  Even  if  a  defending  force  would  win  a 
victory  in  terms  of  tank  exchange,  a  single  enemy  breaking  through  the  defense  could  kill 
the  objective  target.  A  common  tactic  was  to  send  a  single  tank  against  a  target,  in  the 
hope  that  it  would  be  undefended  or  if  defended  that  the  attacker  could  kill  the  objective 
before  being  killed.  Because  of  the  advantage  of  massed  forces  (discussed  in  Section 
C.l),  a  force  would  almost  never  attempt  to  defend  all  of  its  targets. 

3.  The  Sensor  Field  Coverage 

When  it  was  used,  the  sensor  field  covered  only  the  Blue  side  of  the  terrain  and 
thus  was  only  helpful  for  defense.  However,  the  battlefield  geometry  was  such  that  the 
targets  on  the  Blue  side  were  widely  spaced  compared  to  their  distance  from  the  Blue- 
Red  dividing  line,  so  that  the  sensors  did  not  provide  enough  warning  for  a  centralized 
Blue  force  to  engage  an  invading  Red  force  before  it  reached  a  target. 

C.  COMBAT  CHARACTERISTICS 

Many  combat  capabilities  affected  battle  outcomes,  e.g.,  weapon  effectiveness, 
detection  range,  etc.  The  characteristics  discussed  in  this  section  are  those  identified  as 
affecting  tactical  choices.  One  reason  for  this  discussion  is  that  these  characteristics  may 
in  part  be  artifacts  of  the  simulation,  scenario,  or  test  design,  so  that  they  may  have 
affected  tactical  choices  in  unrealistic  ways. 

1.  Massing  Forces 

A  numerical  superior  force  had  a  greater  advantage  than  might  be  expected.  No 
specific  tests  were  made,  but  as  an  estimate  two  platoons  (eight  tanks)  might  be  expected 
to  defeat  one  platoon  (four  tanks),  killing  all  four  of  the  enemy  while  losing  no  more  than 
one  or  two  tanks.  Battles  happened  very  quickly,  so  that  a  disorganized  force  that  did  not 
get  all  of  its  tanks  into  the  battle  at  the  same  time  was  at  a  severe  disadvantage. 

2.  Defending 

Arranging  tanks  in  good  defensive  positions  was  very  much  a  guessing  game. 
There  seemed  to  be  two  factors  involved:  intervisibility  and  motion. 
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Intervisibility  was  the  more  important  factor,  particularly  in  hilly  terrain.  Tanks 
placed  near  a  defended  target  might  not  see  attackers  until  after  the  attackers  saw  the 
target,  or  a  group  of  attackers  might  see  the  defenders  one  at  a  time  and  have  a  numerical 
advantage  over  each.  The  simulation  environment  provided  no  good  way  to  quickly  find 
partially  concealed  positions  with  good  sight  lines.  Presumably,  this  was  an  important 
reason  that  placing  defenders  very  close  to  the  defended  target  was  a  good  tactic. 

Overall,  the  estimate  was  that  a  moving  attacking  force  has  a  slight  advantage 
over  a  stationary  defensive  force,  and  operators  would  generally  move  the  defending 
force  towards  the  attackers  during  combat. 

D.  SUMMARY  OF  FAVORED  TACTICS 

A  number  of  elements  were  commonly  used  as  parts  of  a  strategy  for  any  given 
run,  including: 

•  Emphasis  on  Attack — A  consistent  emphasis  was  on  attack  throughout  the 
trials.  Defense  was  difficult,  and  even  a  well-conducted  defense  might  result 
in  the  loss  of  a  defended  target  before  the  attackers  would  all  be  killed.  There 
was  also  a  kind  of  asymmetry  in  the  test  setup  that  inclined  the  operators  to 
attack.  A  game  could  not  be  won  without  attacking  targets  in  enemy  territory. 
And,  although  the  game  could  not  be  won  without  a  friendly  target  surviving, 
this  did  not  necessarily  mean  that  the  targets  needed  to  be  defended.  In 
particular,  if  an  attacking  force  met  and  defeated  an  enemy  attacking  force,  the 
enemy  might  be  unable  to  kill  all  friendly  targets  with  his  remaining  tanks, 
within  the  time  limit. 

•  Massed  Forces — A  larger  force  would  consistently  defeat  a  smaller  force, 
with  a  favorable  exchange  ratio  that  left  it  able  to  defeat  a  subsequently 
encountered  smaller  force. 

•  Selective  Defense — The  perceived  need  to  mass  forces  and  to  attack  led  to  a 
resource  allocation  problem:  insufficient  forces  to  mass  defenders  at  each 
site.  In  many  cases,  some  or  even  all  friendly  target  sites  were  left 
undefended. 

•  Attack  with  Individual  Tanks — While  a  numerical  superiority  was  a  great 
advantage  for  tank  encounters,  only  a  single  tank  was  required  to  defeat  an 
undefended  enemy  target.  A  common  tactic  was  to  send  individual  tanks  to 
attack  targets,  in  case  they  were  not  defended. 
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1.  Tactics  in  the  First  Trial  Phase 

In  this  phase  the  two  commanders  for  a  given  side  seldom  tried  to  unify  their 
forces,  and  most  runs  became  separate  battles  to  the  east  and  west  of  the  terrain.  There 
were  variations  in  the  attack  route  and  in  the  size  of  the  attacking  force. 

A  tactic  that  became  common  for  Red  to  use  when  Blue  had  a  situational 
awareness  advantage  was  to  position  his  forces  near  the  center  of  the  terrain  and  to  attack 
with  all  of  them  with  about  20  minutes  left  in  the  run.  This  conceded  the  loss  of  all  his 
targets  but  usually  resulted  in  killing  all  Blue  targets.  The  thought  was  that  a  tie  was 
sufficient  when  facing  the  situational  awareness  disadvantage. 

2.  Tactics  in  the  Second  Trial  Phase 

The  tactics  used  in  the  second  phase  were  more  varied  than  those  in  the  first.  In 
most  runs  substantial  cooperation  occurred  between  the  two  commanders  for  a  side.  It 
was  most  common  for  the  western  commander  to  send  a  platoon  to  join  an  attack  in  the 
east.  The  attack  might  include  one  or  both  eastern  platoons.  In  some  cases,  the  other 
western  platoon  would  cross  to  defend  the  eastern  targets.  Less  common  were  cases 
when  a  unified  force  would  attack  near  the  center  or  in  the  western  parts  of  the  terrain. 
Near  the  end  of  the  set  a  few  cases  occurred  in  which  a  side  attacked  with  all  four 
platoons  in  the  east-central  area.  The  thought  was  that,  after  defeating  the  opposing  force 
in  this  area,  some  of  the  attacking  force  would  drop  back  to  defend  the  eastern  targets  and 
the  rest  would  continue  to  the  opposing  targets. 
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V.  CONCLUSIONS  AND  RECOMMENDATIONS 


This  experiment  did  not  demonstrate  that  having  an  advantage  with  respect  to 
situational  awareness  yields  a  significant  benefit  in  small  unit  armor  engagements.  As 
this  report  suggests,  time  constraints  and  artifacts  of  the  simulation  may  have  driven 
outcomes.  Chapter  IV  outlines  these  issues  and  discusses  their  impact  on  the  conduct  of 
the  games.  If  this  is  indeed  the  case,  then  redesigning  the  experiment  with  these  factors 
in  mind,  and  possibly  selecting  a  different  simulation  (e.g.,  JCATS),  might  show  different 
results. 

On  the  other  hand,  one  could  argue  that  sufficiently  many  replications  of  the 
experiment  were  conducted  to  "average  out"  random  anomalous  behavior  in  the 
simulation.  Also,  with  so  many  replications,  consistent  artifacts  would  likely  have  been 
recognized  and  successfully  "gamed"  by  the  players.  If  this  is  indeed  the  case,  then  the 
experiment  calls  into  question  the  value  of  enhanced  situational  awareness.  Continuing 
this  line  of  argument,  the  experiment  may  be  demonstrating  that  either: 

•  The  SA  advantage  must  be  overwhelming  (such  as  one  side  having  complete 
knowledge  of  its  opponent’s  position  and  strength). 

•  There  is  little  benefit  from  the  information  provided  by  enhanced  SA  without 
the  corresponding  experience  and  discipline  to  make  decisions  based  upon  the 
additional  information. 

The  time  and  resources  available  to  this  CRP  do  not  permit  the  development  of 
either  of  these  hypotheses.  Future  research  involving  experiments  of  the  elementary  type 
conducted  here  might  shed  light  on  the  first,  however.  The  second  is  arguably  more 
subtle  and  may  require  a  level  of  insight  into  tactics  and  procedures  only  found  in 
individuals  with  extensive  military  experience.  More  sophisticated  experiments 
involving  individuals  with  the  relevant  training  are  probably  necessary  to  successfully 
address  the  second  hypothesis. 
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Table  A-1.  Phase  1  A— Tests  of  Between-Subjects  Effects 


Source 

Dependent 

Sum  of 

df 

Mean 

F 

Variable 

Squares 

Square 

Corrected  Model 

XRATIO 

7.588 

9 

0.843 

1.079 

SCORE 

5.479 

9 

0.609 

0.865 

Intercept 

XRATIO 

4.387E-02 

1 

4.387E-02 

0.056 

SCORE 

1.081 

1 

1.081 

1.535 

VIZ*  SENS 

XRATIO 

1 .493E-02 

1 

1.493E-02 

0.019 

SCORE 

9.393E-02 

1 

9.393E-02 

0.133 

VIZ 

XRATIO 

0.978 

1 

0.978 

1.252 

SCORE 

0.667 

1 

0.667 

0.947 

SENS 

XRATIO 

0.604 

1 

0.604 

0.773 

SCORE 

1.748 

1 

1.748 

2.483 

TEAM 

XRATIO 

5.254 

5 

1.051 

1.345 

SCORE 

2.616 

5 

0.523 

0.743 

RUN  ORDER 

XRATIO 

0.649 

1 

0.649 

0.831 

SCORE 

1.813 

1 

1.813 

2.575 

Error 

XRATIO 

10.939 

14 

0.781 

SCORE 

9.854 

14 

0.704 

Total 

XRATIO 

51.307 

24 

SCORE 

18.000 

24 

Corrected  Total 

XRATIO 

18.527 

23 

SCORE 

15.333 

23 

Ratio  R  Squared  =  0.410  (Adjusted  R  Squared  =  0.030) 


Score  R  Squared  =  0.357  (Adjusted  R  Squared  =  -0.056) 
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Sig. 

0.433 

0.575 

0.816 

0.236 

0.892 

0.720 

0.282 

0.347 

0.394 

0.137 

0.302 

0.604 

0.377 

0.131 


Table  A-2.  Phase  I  B— Tests  of  Between-Subjects  Effects 


Source 

Dependent 

Sum  of 

df 

Mean 

F 

Sig. 

Variable 

Squares 

Square 

Corrected  Model 

SCORE 

4.425 

9 

0.492 

0.615 

0.766 

XRATIO 

7.368 

9 

0.819 

1.028 

0.465 

Intercept 

SCORE 

0.283 

1 

0.283 

0.353 

0.562 

XRATIO 

3.824 

1 

3.824 

4.803 

0.046 

VIZ  *  SENS 

SCORE 

1.024 

1 

1.024 

1.280 

0.277 

XRATIO 

3.931  E-02 

1 

3.931  E-02 

0.049 

0.827 

VIZ 

SCORE 

0.375 

1 

0.375 

0.469 

0.505 

XRATIO 

0.997 

1 

0.997 

1.252 

0.282 

SENS 

SCORE 

1.058 

1 

1.058 

1.322 

0.269 

XRATIO 

0.540 

1 

0.540 

0.678 

0.424 

TEAM 

SCORE 

1.486 

5 

0.297 

0.371 

0.860 

XRATIO 

4.434 

5 

0.887 

1.114 

0.397 

RUN  ORDER 

SCORE 

9.135E-02 

1 

9.1 35  E-02 

0.114 

0.740 

XRATIO 

0.385 

1 

0.385 

0.483 

0.498 

Error 

SCORE 

11.200 

14 

0.800 

XRATIO 

11.149 

14 

0.796 

Total 

SCORE 

19.000 

24 

XRATIO 

51 .555 

24 

Corrected  Total 

SCORE 

15.625 

23 

XRATIO 

18.517 

23 

Score  R  Squared  =  0.283  (Adjusted  R  Squared  =  -0.178) 
Ratio  R  Squared  =  0.398  (Adjusted  R  Squared  =  0.01 1) 
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Table  A-3.  Phase  II.  Tests  of  Between-Subjects  Effects 


Source 

Dependent 

Sum  of 

df 

Mean 

F 

Sig. 

Variable 

Squares 

Square 

Corrected  Model 

SCORE 

15.063 

9 

1.674 

0.572 

0.798 

XRATIO 

2.249 

9 

0.250 

1.352 

0.296 

Intercept 

SCORE 

1.838 

1 

1.838 

0.629 

0.441 

XRATIO 

2.101 

1 

2.101 

11.371 

0.005 

VIZ  *  SENS 

SCORE 

5.474 

1 

5.474 

1.872 

0.193 

XRATIO 

5.153E-03 

1 

5.153E-03 

0.028 

0.870 

VIZ 

SCORE 

1.251 

1 

1.251 

0.428 

0.524 

XRATIO 

4.149E-02 

1 

4.149E-02 

0.225 

0.643 

SENS 

SCORE 

0.167 

1 

0.167 

0.057 

0.815 

XRATIO 

7.019E-06 

1 

7.019E-06 

0.000 

0.995 

TEAM 

SCORE 

4.951 

5 

0.990 

0.339 

0.881 

XRATIO 

1.906 

5 

0.381 

2.062 

0.131 

RUN  ORDER 

SCORE 

1.896 

1 

1.896 

0.648 

0.434 

XRATIO 

0.251 

1 

0.251 

1.356 

0.264 

Error 

SCORE 

40.937 

14 

2.924 

XRATIO 

2.587 

14 

0.185 

Total 

SCORE 

56.000 

24 

XRATIO 

34.675 

24 

Corrected  Total 

SCORE 

56.000 

23 

XRATIO 

4.836 

23 

Score  R  Squared  = 

■  0.269  (Adjusted  R 

.  Squared  = 

-0.201) 

Ratio  R  Squared  = 

0.465  (Adjusted  R  Squared  = 

0.121) 
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ANOVA 

analysis  of  variance 

C2 

command  and  control 

CMS 

Comprehensive  Mine  Simulator 

CRP 

Central  Research  Project 

DIS 

Distributed  Interactive  Simulation 

HITL 

human  in  the  loop 

HMMWV 

high-mobility  multi-wheeled  vehicle 

km 

kilometer 

kph 

kilometers  per  hour 

ModSAF 

Modular  Semi- Automated  Forces 

PDU 

protocol  data  unit 

PVD 

plan-view  display 

SA 

situational  awareness 

SAF 

Semi- Automated  Forces 

SIMCTR 

Simulation  Center  (at  IDA) 

TFXXI-AWE 

Task  Force  XXI  Advanced  Warfighting  Experiment 
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